i-0026e1085400d6372)I set up my EC2 instance with the following configurations:
de4-week3 AMI:t3.small instance type:gergely-week2 IAM role:patrik-szigeti-assignment security group:patrik-szigeti-assignment-key newly created key pair:patrik-szigeti-assignment-instance, and its Instance ID is i-0026e1085400d6372:The instance’s setup was successful, logging in to RStudio with username ceu and password ceudata:
I set up an app.properties file in the streamer folder, set the stream name to CEU-Binance, the region to Ireland (eu-west-1) and named my application patrik-szigeti-assignment-app:
I created app.R based on the file from the lecture with the following modifications:
symbol, I also had to extract the price (named p in the stream) and the quantity (q) for each transaction so that I could calculate amount by multiplying the two.processed_data) before the for-loop for the records in the batch.symbol and amount to this data table.symbol to get the overall traded amount for each symbol in the batch.redisIncr to store the aggregated values in Redis.app.log logs all incoming batches with additional information about the received data:
I created a new Jenkins job as a Freestyle project under the name CEU-Binance Assignment that I later renamed to ceu-binance-assignment to keep consistency with the naming convention I applied so far.
The script being called from Jenkins is the overall_value.R file which connects to Redis and creates a list from the key-value pairs that are stored in the Redis cache. As these items are coming through as a list, I have to unnest the values with lapply, and then cast them as.numeric. After dropping the original list, I’m able to aggregate the column and calculate the sum of the transactions in the last batch. The overall_transaction_value function takes care of the data processing and transformation, and finally prints out the overall value of the transaction in USD.
Unfortunately I kept running into a “file not found” issue in Jenkins when trying to run Rscript /home/overall_value.R, so I ended up pasting the whole code snippet into “Execute shell command”:
The job ran successfully (after a couple of tries, this was run #20), and returned the overall value for the last batch:
I created two plots to post to #bots-final-project, but before that, I had to create a data table from the key-value pairs, and then transform the amounts that were coming through as a list to numeric values. I also cleaned up the symbols column a bit, and removed the amount: prefix that I used to store the values in Redis.
BTCUSDT, since otherwise that would take up a significant area from the chart, and I wanted to look at the “smaller guys”. For this, I also had to install the treemapify library to the server.I posted both charts to the Slack channel:
And I didn’t forget to stop my instance either: